Aggregating Forecasts of Chance from Incoherent and Abstaining Experts

Linear averaging is a popular method for combining forecasts of chance, but it is of limited use in the context of incoherent or abstaining judges. Recently proposed, the coherent approximation principle (CAP) generalizes linear averaging to have wider applicability yet suffers from computational intractability in cases of interest. This paper proposes a unified framework that views CAP and linear averaging as aggregation methods lying at opposite extremes of a speed-coherence trade off, and suggests a principled methodology for compromise. Exploiting the logical simplicity of events typically assessed by human judges, an aggregation tool (SAA) is developed that offers an acceptable level of coherence in practical amounts of time. SAA enjoys provable performance guarantees, and in several experiments is shown to offer both computational efficiency and competitive forecasting gains. SAA is relevant to the practice of decision analysis, for it enables an elicitation methodology in which judges have freedom to choose the events they assess.